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lBSTRACT 



Two measures of effectiveness for university 
.ibraries are constructed. The Primary Measure of Effectiveness, 
lefined over the total population of users, considers the number of 
naterial and information items desired by the users, the number of 
satisfactorily answered inquiries, and the number of users. For the 
Secondary Measure of Effectiveness the total population is classified 
Into subsets according to similarity of users* needs. A measure is 
lefined for each subset and is a function of the number of 
Individuals in the subset. (Author) 
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Previous papers have argued that in order to describe, design, or evaluate 
a university library system, it is not sufficient to consider the library as a 
separate entity. The behavior and utility of a university library is constrained 
and 'influenced by actions of the funders and users of the library. Several specific 
studies have been reported including the development of a conceptual systems 
model (2) , the construction and analysis of an industrial dynamics simulation 
model of university departmental libraries (3,10), and the formulation and analysis 
of mathematical models of optimal user search behavior (1). However, to this point 
no adequate measures of effectiveness exist which could be utilized by university 
libraries in evaluating on-going programs ex in determining the impact of proposed 
programs . 

The purpose of this paper is to propose two such measures and to discuss , in 
detail, the properties of each measure. A large number of user studies have been 
conducted (5) and the conclusion can be drawn that a measure of effectiveness 
must adequately reflect satisfaction of user needs. The "primary measure of 
effectiveness" is constructed to accomplish precisely this aspect. 

A recent empirical study (12) conducted at Purdue University demonstrates that 
the total population of potential library users can classified into subsets 

faculty , graduate students, and undergraduate students— —according to similarity of 
needs. A "secondary measure of effectiveness" is defined which is appropriate for 
each subset and which considers number of actual users and number of repeat users. 
Library Goals and User Needs 

Phe primary goals of the library are defined to be: 

1. Maximize user need satisfaction. 

2. Minimize the time loss (opportunity cost) to the user. 

In addition, the university seeks to serve as many users as feasible. Thus, a 
secondary goal is: 



3. Increase the number of actual users. 

These goals serve as foundations for both measures of effectiveness. 

Despite work by Bourne, et al (4) and Meier (8), adequate methodologies 
for collecting data on time loss do not exist. However, their results suggest 
that time loss can be accounted for indirectly through consideration of the 
increase (decrease) of the number of users and of the Increase (decrease) in the 
use of library materials . 

A questionnaire designed by Dean R. B. Downs, the Dean of Library Administration 
at the University of Illinois and by Dr. Warren Seibert of the Purdue Libraries 
was recently administered at Purdue University (6) . Question 5 asked users to 
identify their "primary reason for coming to the library" from a list of 11 
possible reasons including "do something else (not mentioned above)." The major 
reasons cited are: 

1. Find and read material required for a course. 

2. Read material for self improvement. 

3. Read for pleasure or fun. 

4. Borrow library material for further reading. 

5. Do research for a term paper. 

6. Do research for graduate exams or thesis work. 

7. Do research for a publishable paper or book. 

8. Get some material copied (Xerox). 

9. Return materials (books) to the library. 

10. Do homework with own books. 

11. Seek information which does not require borrowing of library materials 
(questions answered by reference librarians). 

Statistical methodologies for measuring library usage, e.g., re-shelving and 
browsing statistics, have been investigated in depth by Jain (7). These approaches 
could be utili 2 ed to obtain information on reasons 1-7 . The number of persons 
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getting material copied, and the number of copies reproduced, are routinely kept 
now by many libraries. The only difficulty likely to be encountered relative to 
item 9 is the identification of users who came to the library only for the purpose 
of returning material. Sampling studies would be necessary to accompli-h this 
identification and to collect information on item 10. The total number of requests 
for information could be easily recorded by the reference librarians. A much more 
difficult item to obtain is whether '* - inquiry was satisfactorily answered. This 
may be a purely perceptual aspect which can be provided only by the user or by 
some panel of experts. Regardless, the assumption is that data relative to user 
need satisfaction can be collected for the evaluation of implemented programs. 

Exactly the same information would be required in order to evaluate benefits 
likely to accrue from proposed programs. If pilot programs were conducted, then 
the same methodologies could be utilized. Otherwise, subjective estimates would 
be required. Specific approaches are not defined; however, research is underway 
investigating the applicability of procedures such as those suggested by Moore and 
Baker (9). 

A Primary Measure of Effectiveness 

In order to specify a primary measure of effectiveness, three specific mea- 
sures will be constructed and integrated. The three measures are concerned with 
1) the total number of users and the total number of material items used, 2) the 
total number of non-material information items sought and the total number supplied 
and 3) the total number of users who are studying with their own materials or 
who are at the library for social purposes. 

The following definitions will be utilized in the development of a Primary 

Measure of Effectiveness: 

N = total population of users of the university library, e.g., all students, 
faculty, alumni, and staff associated with the university. 
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n 



= total number of users from N who use the library facilities per time 



period. Note that n ^ N. 

r, = total number of non-material information items sought per time period. 



e.g. , the total number of questions asked of reference librarians. 
r 2 = total number of acceptable non-material information items supplied per 



time period, e.g., the total number of questions answered satisfactorily 
by reference librarians . 

s = total number of users per time period who are studying with their own 
materials or who are there for social purposes . 

m — total number of material items used (reshelved) per time period, e.g. , 
the material used in order to satisfy the needs underlying items 1—8. 
Effectiveness Term For m And n 

The material items accessed during a time period are selected because of their 
potential, as perceived by the user, to satisfy some underlying need. Effectiveness 
of the library operation is a function of the total number of items accessed 
(m 1 ) and the total number which actually contributed to need satisfaction (m 2 > , 
e.g., ra 2 should be maximized and (m^-m 2 ) shou 1 A , be mtu/ §vu'/Vacf to the con- 

traint that m 1 >_ m ? . an effectiveness term such as mo ^ m 2 ^ would accomplish the 
above; an : indicates that m 2 is increasing air that ( m i“ m 2 ^ is 



is made. Assume that any item which is removed from its shelf — Location during a 
time period, and hence reshelved, is sufficiently relevant to crantribute to need 
satisfaction,. The total number of items reshelved is denoted fry ai and, as na 
increases , the effectiveness of the library increases. 

The secondary goal suggested in the previous section is tc increase the number 
of actual users. The total number of actual users is denoted fey n and as n 
^-increases, rhe library's effectiveness increases. The relative impact of both m 
EJM£ id n should- be related to N, the total population of potential library users. 



decreasing 



lighfr^of the difficulty inherent in measuring m 2> a simplifying assumption 
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The following term for measuring the effectiveness of the library relative to 
satisfying user material needs and to the number of actual users is proposed: 




Table 1 summarizes the properties of e^ under various possible conditions of change 
resulting from modifications in the library’s operations. 

In summary, given e^ as defined, the following conclusions can be drawn: 

1. As the number of material Items used during a time period increases 
(decreases) , e^ increases (decreases) . 

2. As the number of users during a time period increases (decreases), e^ 
increases (decreases) . 

3. If the number of material items used is less (greater) than the sum of the 
total population plus the number of users, then increasing the number of 
material uses is more (less) desirable than increasing the number of users. 

4. If no material Items are being used, e^ is equal to zero regardless of the 
number of users. If the users are coming for some need not. reT-ca 
accessing library material, the effectiveness term for r^ and r 2 » or the 
the term for s, will measure the associated contribution to library 
effectiveness . 

If e^ is accepted, then. the above conditions follow; conversely. If the above 
conditions are acceptable, then e^ is an acceptable effectiveness term for relating 
the number of material items used with the number of users. 

Effectiveness Term For r^ And 



The effectiveness of a university library should increase as the number 
of non-material information items supplied per time period (r«) increases. The 
effectiveness should also increase, but to a lesser degree, as the number of 
non-material information items sought (r^) increases. An increase in the number 
sought implies that the users perceive an improvement in the library's ability 
to satisfy non-material information needs. Over time, however, unless ability to 
O 
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Table 1 
Properties of 



Conditions of Change 


Impact on e x = §- (l+ 

A S ( i+ s) 




i. 


Am ^ 0, An = 0 






a) Am >0 


Ae^> 0 






b) Am <0 


Ae 1 < 0 




2. 


Am = 0, An ^ 0 


t _ _ m(An) 

Ae - 2 

N 






a) An >0 


ie i> 0 






b) An <0 


o 

V 

H 

<1> . 
< 




3. 


Am ^ 0, An ^ 0 


Ae., = [Am(N) + Am(n) + An(m) + 


An (Am) ] 






N 






a) Am >0, An >0 


Ae^> 0 






b) Am <0, An <0 


Ae^< 0 






b-1) |An| = n 


Ae^ = -e 1 (and |Am 


| » m) 




b-2) |An| < n,|Am| = m 


Ae l " ~ e l 






c) Am> 0, An< 0 


^Tn > ° 






c-1) Am = | An | 


Am <N+n - m => Ae^> 


0 




d) Am <0, An >0 


, . v Am m + Am 

N + n _> ie l' 


> 0 




d-1) An = | Am | 


An< m- (N+n) =>Ae^> 


0 



In the above table, the following conditions hold: 
i) m >0* 0 <n <N, N >0. 



±i) If Am <0, |Am|j< m 
iii) If An >0, An _<N-n 
iv) If ^n <0, |An|^c n. 
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satisfy these needs also improves, it is reasonable to expect the number of non- 
material information items sought to stablize or even decrease. Thus, r ^ should 
be the dominant consideration. 

The recommended effectiveness term is: 

( 2 ) 






Table 2 summarizes the properties of under various possible changes in r^ 
and r^. 

As suggested in the above discussion, increase (decrease) in the number of 
non-material information items sought or increase (decrease) in items supplied 
produces an increase (decrease) in e 2 » In addition, since + Ar£< r^ + Ar^ , 
i is always advantageous to increase the number of items supplied, even if 
the number of items sought is undergoing an equal decrease. However, in the long- 
run, increase in would be expected to generate additional items sought, i.e., 

increase r^. 

Effectiveness Term For s 

The effectiveness of the university library should increase (decrease) if its 
facilities are used by a greater (lesser) number of individuals for study or social 
purposes (s). In this case, the underlying user need is a need for space rather than 
a need for material or non— material information items. The following term exhibits 
the desired properties: 



e 3 



(3) 



N 



and Ae, = — 

J N 



The denominator N z is suggested, in order to have changes 



in s comparable to changes in n and r^ since all three terms reflect the number 
of users utilizing the library. If the denominator were to be specified as N, 
then s would be considered as on the same level as n and r_ both of which re- 
present satisfaction of an information need. 






8 



8 



Table 2 



Properties of e 7 



Conditions of Change 



Impact on e^ 



3EEy 



Ar 2 


* 0 , 


Ar l 


a) 


Ar 2 


>0 


b) 


Ar 2 


<0 


Ar 2 


= 0 . 


> Ar 


a) 


Ar l 


>0 


b> 


Ar l 


<0 



4r 2 t ir 2 ( r l? 

Ae,. ~ r: + 



'2 N 



N 



Ae 2 > 0 
Ae 2 < ° 



Ae 2 = 



Ar l ^ r 7} 



N 

Ae 2 > 0 
Ae 2 < 0 



3. Ar 2 + 0, Ar x ^ 0 



Ae„ = ~ ~ [N(Ar 0 ) + r 2 (Ar-^H- ^(Ar^ 

+Ar 1 (Ar 2 >] 



a) 


Ar 2 


V 

0 

w 


Ar^ 


0 


b) 


Ar 2 


<0, 


A r x < 


0 


c) 


Ar 2 


>0, 


Ar l" 


0 



:-l) ^r 2 =1^1 



Ae 2 > 0 
Ae 2* 0 



Ar 



(-D 



9 Ar 2 n 

2 > _£_ —— =>Ae 2 > 0 



Ar x N + r, 
Ae 2 > 0 



d) Ar 2 < 6, Ar 1 >0 

d-1) Ar^ =|Ar 2 | 



(- 1 ) 



Ar 2 < r 2 + Ar 2 



Ar, 



N + r. 



=>Ae„ >0 



Ae 2 <0 



In the above table, the following conditions hold: 
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i) 


0 < r 2 < r l 


ii) 


If Ar 2 <0, 


iii) 


If Ar x <0, 



iv) r 2 + Ar 2 - r l + Ar l 
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The Primary Measure of Effectiveness 

The primary measure of effectiveness, E, is defined as a weighted sum of 
e l* e 2* ant * e 3‘ Specifically, 



E = w^ e^ + e 2 + (4) 

where w^, w 2 , and are the weighting factors. 

The weighting factors allow the library administration to designate the relative 
importance- of the contribution of each of the terms. Although there exist several 
approaches for determining v^*,w 2 , and w^ (e.g. , 13), experience at Purdue University 
suggests another procedure which will be illustrated by the following example. 

Suppose that the university library administrator states that an increase 
of 20 patrons studying in the library (As) is equivalent (in value) to an increase 
of 1 question answered properly (Ar 2 ) and that an increase of 3 material items 
(Am) is also equivalent to Ar 2 =1. Thus, it follows that 



W 3^ e 3^ 



w 3 (As) 




20 w 3 w 2 (N+r 1 ) 




w» 




M 2 J 



- 



(ie 2 ) 



and that 



t ' 

= w i [% 



(1 + " 



3w i 



3w- (N + n) 



If w, is arbitrarily set equal to 1, then 
2 N + r. 

w_ = and w 1 = 

J 20 



N + r. 
3 (N+n) 




(N+r 1 ) 

2 



Note that if the administrator is consistent in stating his preference, then 
Am - 3 is equivalent to As = 20. 
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A Secondary Measure of Effectiveness 

The recognition of user subsets with dissimilar needs is considered important 
for the direction of program development, e.g. , one type of program might benefit 
one subset and not contribute at all to another subset. Clearly, E could 
be utilized to study any given subset provided the input data is available. However, 
a measure is proposed which is different in form from E and is based on the number 
of users and the number of times the library is accessed by the users. 

The division of the user population should be according to user needs. Rzasa 
and Moriarty (12) established one such division as faculty, graduate students, 

undergraduate students, and staff and others. Another division might be according 
to the formal organization structure, e.g., by the academic departments. The point 
is that the division is flexible and should be based on the specific environment 
under study. Let J be the number of subsets and let j be a subscript over the 
subsets, j=l,2. ..., J. 

The secondary measure of effectiveness reflects the following properties : 

1. As the number of individuals from a subset who use the library in a time 
period ( n j) increases, the measure should also increase. 

2. As the number of times users from a subset use the library facilities 



3. 



within a time period ( u j) increases 



the measure should also increase. 



If the library is serving a largse percentage of subset j, but the 
number of return visits is low, it is preferred that a lower percentage 



be served and that the users return more often. 



4. If the library is serving a small percentage of subset j, but the number 
of return visits is high, it is preferred that a higher percentage be 



served even though the users return less often. 



A secondary measure of the form below satisfies the above conditions! 

< 5 ) 



o 
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where A. = u. - n and is defined as "activity” and N. is the total population of 
J j J J 



subset j . Ej is proposed to measure changes in effectiveness relative to subset 



j and is not intended to compare effectiveness relative to subset j to effectiveness 
relative to some other subset j’, i.e., a comparison of to t is not suggested. 



It might be noted that a secondary measure of effectiveness, E, can be related 
to the entire population which the library serves . E will give an indication of 
effectiveness based on the total number of users from the population, N, and the 
number of times the users return to the library during a given time period. The 
library administration will thus have an indication of how their facility rates 
as a source of need satisfaction by the users. 

Specifically, the secondary measure of effectiveness should aid the administrate: 
in evaluating programs aimed at a specific group. Table 3 summarizes the properties 



of E. under various possible conditions of change resulting from modifications 



in the library’s operation. 



In summary, given E^ as defined, the following conclusions can be drawn: 



0 
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1. If activity increases (decreases) while the number of users remains con- 
stant, effectiveness increases (decreases) since the number of users 
from group j are returning more (less) often. 

2. Activity remaining constant while the number of users, iij , increases is 
equivalent to saying that the new users are only coming once during a 
time period. Since the library is serving a larger proportion of the 
subpopulation, effectiveness increases. 

3. If both activity and the number of users increase (decrease) then not only 
is a larger (smaller) proportion of group j using the facility but the 
return rate is increasing (decreasing). Therefore, effectiveness increases 
(decreases). 



Table 3 



Condi tions of Change 
1 . A A ^ 0 An^ ~ 0 

J 




a) AA.>Q 

3 

b) AA. <0 

3 

b-1) | AA | = A 

J J 

2. AA = 0, An f- 0 
J J 

a) An , > 0 

j 

b) An.< 0 

3 

b-l) Ian,] = n. 



*Y ° 

*Y 0 





3 . AA^ 0 , An^ 5^0 



a) 


AA. 

3 


>0, 


o 

A 

<1 






b) 


A V 


: 0, 


A V 


0 






c) 




>0, 


An,,< 

J 


0 










Let. 


A 

A J = 


A j 




AA j 








"j ‘ 


n. 

3 




An j 


d) 


AAj 


<0, 


an 3 


>0 








[ (AA.. ) (An^ ) 



AE^> 0 

ae.< 0 

C&.Xn.j) >(A j )(n j ) 
(A.) (n^) >(A j )(n j ) 



In the above table, the following conditions hold 




i) Since A j - n ^ , A .£ ° » i,e ’ U j- n j 

ii) If Au. <0, An,< 0, and AA.> 0 (Case 3c) 

thenjAu. | <. |An^J 

iii) If Au .;> 0, An > 0 and AA.< 0 (Case 3d) 




then Au . 

3 



< An 
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4. The relationship between change in the number of users and change in 

effectiveness is influenced by the magnitude of the percentage of the group 
who use the facility. 

a. If the number of users decreases; then, in order for effectiveness to 
increase, activity must increase. The required level of activity in- 
crease is inversely related t< percentage using the facility, 

b. If the number of users increase s* tbe effectiveness can increase 
even if activity decreases. The lavsl of activity decrease is in- 
versely related to the percentage using tffiB facility. 

/ 

Summary and Comments 

Based upon the goals of a university library and conditions, which, if 
satisfied, led toward attainment of the goals,, t&o measures of effectiveness 
were developed. The criteria established for che primary measure were 
1) the number of material items utilized, 2) the number of actual users, 

3) the number of informational items sought, 4) the number of satisfactory 
information items received, and 5) the amount of study space utilized. The 
total population served as a basis so that changes in the user population could 
be taken into account. 

The criteria established for the secondary measure were 1) the number 
of actual users from a given subpopulation and 2) the return rate of these 
users . The subpopulation which the library served was used as a basis for 
the secondary measure. 

The purpose of this paper was to develop an adequate measure of effectiveness 
fo 7 r university libraries to evaluate on-going programs or in. determining the im- 
pact of proposed programs. Inputs to the model for evaluation of present programs 
can be dete mined by the statistical methodologies msntioned. Inputs for the 
evaluation of proposed programs might be dete israni iied ny the use of pilot studies, 

O bj active estimates by administrators or industrial! dynamics models . Preferred 

ERJC 

. ^•'pr ograms could then be evaluated by using a cost-effectiveness analysis. 



One final comment Is needed. Although the measures developed satisfied 
the conditions which were specified, they are not unique. That is, as long as th 
model developed satisfied specific conditions which are directly related to the 

goals, the measure would be adequate. 
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